Anthropic's Claude 3 Opus has surpassed OpenAI's GPT-4 for the first time on Chatbot Arena. Chatbot Arena is a leaderboard run by the Large Model Systems Organization, a research organization dedicated to open models. Its site allows visitors to rate outputs from various models, enabling it to calculate the best models in aggregate. While Claude's rise is notable, GPT-4 is now over a year old.

Anthropic has trained three new models in the Claude 3 family, the strongest of which matches GPT4’s reported benchmark results. It is also a multimodal model and performs well on vision tasks. Importantly, Claude's coding ability is substantially improved with this release.

Andrej Karpathy issued a long context challenge to make a blog post from a recent video of his. Claude 3 was able to perform this task with some data pre-processing help. The resulting blog post is high quality and interesting.

Anthropic's new AI model, Claude 3, stands out for its 'warmth,' making it a robust partner in creative writing tasks. Claude 3 is described as more human-feeling and naturalistic, crossing the threshold from good thought to deep thought made enjoyable. Despite technical benchmarks not fully capturing this nuance, Claude 3 is poised to revolutionize how we interact with AI in creative processes.

The release of Claude 3 has been quite popular, but the prompting style for these models is slightly different. Anthropic has collected a set of user prompts that work well for a wide variety of tasks and topics.